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The method of buffering packets in a digital 
communications device includes defining an n-level 
hierarchy of memory partitions, wherein each non-top 
level partition has one or more child partitions at an 
immediately lower level of the hierarchy. The memory 
partitions at the top-most level are pre-configured 
with a target memory occupancy size, and the target 
occupancy for each memory partition situated at a lower 
level is recursively computed in a dynamic manner based 
on the aggregate congestion of its parent partition, 
until the target occupancies for the bottom-most memory 
partitions are determined. Each traffic flow which uses 
the buffer is associated with one of the memory 
partitions at the bottom-most level of the hierarchy 
and packet discard is enabled in the event the actual 
memory occupancy of a traffic flow exceeds its target 
occupancy. The memory partitions at higher levels are 
preferably associated with a group or pool of traffic 
flows, such as traffic flows associated with a 
particular egress port and class of service, to thereby 
selectively control aggregate congestion. The method 
allows full buffer sharing, the weighted distribution 
of memory space within a memory partition, and scales 
easily to large systems. 
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Abstract 

The method of buffering packets in a digital conununications device includes 
defining an n-level hierarchy of memory partitions, wherein each non-top level partition has 

5 one or more child partitions at an immediately lower level of the hierarchy. The memory 
partitions at the top-most level are pre-configured with a target memory occupancy size, and 
the target occupancy for each memory partition situated at a lower level is recursively 
computed in a dynamic manner based on the aggregate congestion of its parent partition, until 
the target occupancies for the bottom-most memory partitions are determined. Each traffic 

10 fiow which uses the buffer is associated with one of the memory partitions at the bottom-most 
level of the hierarchy and packet discard is enabled in the event the actual memory occupancy 
of a traffic flow exceeds its target occupancy. The memory partitions at higher levels are 
preferably associated with a group or pool of traffic flows, such as traffic flows associated 
with a particular egress port and class of service, to thereby selectively control aggregate 

15 congestion. The method allows full buffer sharing, the weighted distribution of memory 
space within a memory partition, and scales easily to large systems. 
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BUFFERING SYSTEM EMPLOYING PER TRAFFIC FLOW ACCOUNTING 

CONGESTION CONTROL 

Field of the Invention 



The invention relates to a method and system for bufTering data packets at a 
queuing point in a digital communications device such as a network node. 

Background of the Invention 



In order to effect statistical multiplexing in a store and forward digital 
communications device, such devices wiU typically queue data packets for subsequent 
processing or transmission in a conmion storage resource such as a memory buffer. At such a 
queuing point, the common storage resource may be shared by traffic streams associated with 
15 various quaUty of service classes, interface ports, and aggregate pools or groups of traffic 
flows. With traffic of such a multi-feceted nature, such communication devices will often 
employ some type of congestion control system in order to ensure tfiat tiie common storage 
resource is "fairly** allocated amongst the various traffic streams. 

example, in an asynchronous transfer mode (ATM) communication system, 
tiie most elemental traffic stream is a virtual connection (VC) which may belong to one of a 
number of different types of quaHty of service categories. The ATM Forum Traffic 
Management working group has defined five (5) traffic classes or service categories, which 
are distinguished by tiie parameter sets v^ich describe source behaviour and quality of service 
25 (QoS) guarantees. These categories incUide constant bit rate (GBR), real time variable bit rate 
(rtVBR), non-real time variable bit rate (nrtVBR), available bit rate (ABR), and unspecified 
bit rate (UBR) service categories. The ABR and UBR service categories arc intended to carry 
data traffic which has no ^cific cell loss or delay guarantees. UBR service does not specify 
traffic related guarantees while ABR services attempts to provide a minimum useable 
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band width, designated as a minimuin cell rate (MCR). The ATM Forum Traffic Management 
working group and International Telecommunications Union (ITU) have also proposed a new 
service category, referred to as guaranteed frame rate (GFR). GFR is intended to provide 
service similar to UBR but with a guaranteed minimum useable bandwidth at the frame level, 
which is mapped to the cell level by an MCR guarantee. 

In an ATM device such as a network switch the memory bufTer at any given 
queuing point may be organized into a plural number of queues wliich may hold data packets 
in aggregate for VCs associated with one of the service categories. Alternatively, each queue 
may be dedicated to a particular VC. Regardless of the queuing structure, each VC 
represents a traffic flow and groups of VCs, spanning one or more queues, can be considered 
as "traffic flow sets". For instance, VCs associated with a particular service class or 
input/output port represent a traffic flow set. When tiie memory buffer becomes congested, it 
may be desirable to apportion its use amongst service categories, and amongst various traffic 
flow sets and the individual traffic flows thereof. In particular, in a network >dieie GFR and 
ABR connections are contending for buffer space, it may be desired to achieve a feir 
distribution of the memory buffer between these service categories and between the individual 
traffic flows or groups thereof. 

There are a number of prior art fair buffer allocation (FBA) schemes. One 
scheme for feirly allocating buffer space is to selectively discard packets based on policing. 
For an example of this scheme in an ATM environment, a packet is tagged (i.e., its CLP field 
is set to 1) if the corresponding connection exceeds its MCR, and when congestion occurs, 
discard priority is given to packets having a ceU loss priority (CLP) field set to zero over 
packets having a CLP field set to one. See ATM Forum Technical Committee, (Traffic 
Management working group living list)", ATM Forum, btd-tm-01 .02, July 1998. This 
scheme, however, fails to fairly distribute unused buffer space between connections. 
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Another known scheme is based on multiple buffer fill level thresholds where a 
shared buffer is partitioned with these thresholds. In this scheme, packet discard occurs when 
the queue occupancy crosses one of the thresholds and the connection has exceeded its fair 
share of the buffer. The fair buffer share of a connection is calculated based on die MCR 
value of the connection and die sum of the MCRs of all active connections utilizing the shared 
buffer. However, this technique does not provide an MCR proportional share of the buffer 
because idle 0-^., allocated but not used) buffer, which can be defined as 

^max(0, ^ -Q), where Qs is the buffer fill level, Q is the buffer segment count 
for a connection /, and _ is tiie fair share of buffer aUocated to the connection, is 



10 distributed at random between the connections. 



Another scheme for fairly allocating buffer space through selective discard is 
based on dynanuc per-VC thresholds. See Choudhury, A.K., and Hahne, E.L,, ^'Dynamic 
Queue Length ThreshoU in a Shared Memory ATM Switch", Proceedings of I.E.E.E. 

15 Infocom 96, March 1996, pages 679 to 686, In this schraie the threshold associated with 
each VC is periodically upgraded based on the unused buffer space and the MCR value of a 
connection. Packet discard occurs when the VC occupancy is greater than the VC threshold. 
This method reserves buffer space to prevent overflows. The amount of reserved buffer space 
depends on the number of active connections. When there is only one active connection, the 

20 buffer is not fully utilized, i.e., full buffer sharing is not allowed. 

The above-mentioned prior art does not fairly distribute unused buffer space 
between connections or traffic flow groups, and in particular does not provide MCR 
proportional fair share distribution of the buffer. Some prior art FBA schemes also do not 
25 allow for full buffer sharing. Another drawback with some prior art FBA schemes is the feet 
that the issue of multiple traffic flow groups contending for the same buffer resource is not 
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addressed. The invention seeks to overcome or alleviate some or all of these and other prior 
art limitations. 

Summary of the Invention 

5 

One aspect of the invention relates to a m^hod of partitioning a memory buflfw. 
The method involves defining a hierarchy of memory partitions, including at least a top level and 
a bottom level, wherein each non-bottom level memory partition consists of one or more child 
memory partitions. The size of each top-level memory partition is provisioned, and a rKMninal 

10 partition size for the child partitions of a given non-bottom level memory partition is dynamically 
computed based on the congestion of the given memory partition. The size of each child memory 
partition is dynamically computed as a weighted amount of its nominal partition size. These steps 
are iterated in order to dynamically determine the size of each memory partition at each level of 
the hierarchy. TTie memory partitions at the bottom-most level of the hierarchy represent space 

15 allocated to (individual or aggregate) traffic flows, and the size of each bottom-level partition 
represents a memory occupancy threshold for the traffic flow. 

The memory partitions are preferably "soft* as opposed to ''hard" partitions in 
that if the memory space occupied by packets associated with a given partition exceeds the 

20 size of the partition then incoming packets associated with that partition are not automatically 
discarded. In the embodiments described herein, each memory partition represents buffer 
space allocated to a group or set of one or more traffic flows at various levels of granularity. 
For instance, a third level memory partition may be provisioned in respect of all traffic flows 
associated with a particular egress port, and a second level memory partition may be 

25 associated with a subset of those traffic flows which belong to a particular service category. 
Therefore, die size of a given partition can be viewed as a target memory occupancy size for 
the group of trafflc flows corresponding to the given partition. At the lowest level of the 
hierarchy, however, the partition size functions as a threshold on the amount of memory that 
may be occupied by a (individual or aggregate) traffic flow. When this threshold is exceeded, 
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packet discard is enabled. In this manner, aggregate congestion at higher levels percolate 
down through the hierarchy to effect the memory occupancy dircsholds of individual tmffic 
flows. The net result is the fair distribution of unused buffer space between groups of traffic 
flows and the individual members thereof. 

Another aspect of tfie invention relates to a method of buffering data packets. 
The method involves: (a) defining a hierarchy of traffic flow sets, including at least a top level 
and a bottom level, wherein each non-bottom level traffic flow set comprises one or more 
child traffic flow subsets; (b) provisioning a target memory occupancy size for each top-level 
traffic flow set; (c) dynamically determining a target memory occupancy size for each traffic 
flow set having a parent traffic flow set based on a congestion of the parent traffic flow set; 
(d) measuring the actual amount of memory occupied by the packets associated with each 
bottom level traffic flow; and (e) enabling the discard of packets associated witfi a given 
bottom level traffic flow set in the event the actual memory occupancy size of the 
15 corresponding bottom level traf^c flow exceeds the target memory occupancy size thereof. 

In the embodiments described herein, the target memory occupancy size for a 
given traffic flow set is preferably computed by first computing a nominal target occupancy 
size for the child traffic flow sets of a common parent. The target memory occupancy size 

20 for each such child traffic flow is then set to a weighted amount of the nominal target 
occupancy size. The nominal target occupancy size for a given group of duld traffic flow 
sets preferably changes in accordance with a prespecified fiinction in response to the 
congestion of their common parent traffic flow set. For instance, the embodiments described 
herein deploy geometric and decaying exponential functions for computing the nominal target 

25 occupancy based on the congestion of a disparity between the target and measured memory 
occupancy sizes of a traffic flow set. 

In die disclosed embodiments the invention is implemented within the context of 
an ATM communications system. In these embodiments, the comparison specified in step (f) 

20625744.1 
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is carried out prior to or upon reception of the first cell of an ATM adaption layer 5 (AAL5) 
firame in order to effect early packet discard in accordance with the outcome of the 
comparison. 

5 The buffering system of the invention scales well to large systems ^ploying 

many hierarchical levels. This is because there are relatively few state variables associated 
with each hierarchical level. In addition, most computations may be performed in the 
background and lookup tables may be used» thereby minimizing processing requirements on 
time critical packet arrival. The buffering system also enables full buffer sharing, as 
10 discussed by way of an example in greater detail below. 

Brief Descriftton of the Drawings 

The foregoing and other aspects of the invention will become more apparent 
15 fi-om the following description of specific embodiments thereof and the accompanying 
drawings which illustrate, by way of example only, the principles of the invention. In the 
drawings, where like elements feature like reference numerals (and wherein individual 
elements bear unique alphabetical suffixes): 

Fig. 1 is a system block diagram of a conventional switch architecture 
20 illustrating various queuing points therein; 

Fig. 2 is a system block diagram of a buffering system according to a first 
embodiment of the invention employed at one of the queuing points shown in Fig. 1; 

Fig. 3 is a Venn diagram showing how memory is hierarchically partitioned in 
the first embodiment; 

Fig- 4 is a diagram showing the hierarchical partitionment of the memory in die 
first embodiment in tree form; 

Fig. S is a system block diagram of a buffering system according to a second 
embodiment of the invention; 
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Fig. 6 is a diagram showing, in tree fonn, how the memory in the second 
embodiment is hierarchically partitioned; 

Fig. 7 is a diagram showing, in tree form, an alternative approach to the 
hierarchical partitionment of the memory in the second embodiment; 

Rg. 8 is a system block diagram of a buffering system according to a third 
embodiment of the invention; 

Fig. 9 is a hardware block diagram of a portion of the buffering system of the 
third embodiment; and 

Fig. 10 is a diagram showing, in tree form, how the memory of the thiid 
embodiment is hierarchically partitioned. 

Detailed Description of the Preferred Embodiments 

Fig. 1 is a diagram of the architecture of a conventional network switch or 
node 9, an example of which is the 36170™ ATM switch manu&ctured by 
Newbridge Networks Corporation of Kanata, Ontario, Canada. The node 9 comprises a 
pluraUty of ingress and egress line cards llA and 1 IB for interfacing with the networic (not 
shown) via physical interface ports. Ingress line cards 1 1 A are configured to receive packet 
traf&c from the network via ingress ports 19 and transmit packets to a switching core 13 via 
egress ports 20. The switching core 13, as is known in the art, directs each padcet to the 
appropriate egress line cards I IB. These line cards are configured to receive packet traffic 
from the switching core 13 via ingress ports 19 and transmit packets to the network via egress 
ports 20. 

The line cards llA and 1 IB as well as the switching core 13 are each "store 
and forward" devices and hence present a point, QP, vwthin the node 9 wherein packets are 
queued in a memory or buffer for subsequent processing by die device (hereinafter **queuing 
point"). At each queuing point a buffer management system is provided as part of the store 
and forward functionality. 

20G25744.1 
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Fig. 2 shows an example of a buffer management system 10 employed in egress 
line card UB. The system 10 comprises a common storage resource such as a physical 
memory 12, portions of which are allocated, as subsequently discussed, to various tributaries 
or logical trafBc flows, such as VCs, 25 carried by or multiplexed on aggregate input stx^am 
18. A controller such as queue management module (QMM) 16 organizes and manages the 
memory 12 according to a selected queuing scheme. In the illustrated embodiment, for 
example, the QMM 16 employs an aggregate queuing scheme based on service class and 
egress port. More specifically, the QMM 16 organizes the memory 12 into multiple sets 15 
of logical queues 17. In each set 15 there preferably exists one queue in fespect of each 
service class of the communication protocol. For instance, when applied to ATM 
communications, each set 15 may comprise six (6) queues 17 in respect of the CBR, rtVBR, 
nrtVBR, ABR, UBR, and GFR service categories. Alternatively, the packets associated with 
two or more service categories may be stored in a conmion queue in which case there may be 
less than a 1 :1 relationship between queue and service category. In any event, the number of 
sets 15 preferably corresponds to the number of egress ports 20 of the line card IIB, with 
each set of queues holding packets destined for the corresponding egress port. 

Accordingly, as tfie ingress port 19 receives the packets of aggregate input 
20 stream 18, the QMM 16 decides whether to store or discard a given packet based on certain 
criteria described in greater detail below. If a packet is destined to be stored, the QMM 16 
reserves the appropriate amount of memory, associates each packet with the appropriate 
logical queue 17, and stores the packet in the memory 12. In die illustrated 36170™ switch 
the function of matching an inbound packet to a given logical queue 17 is based in part on 
25 header or address information carried by the packet and stored connection configuration 
information, but it will be understood that other switches may employ various other 
mechanisms to provide this capability. Arbiters 22 each multiplex packets from the logical 
queues 17 to their corresponding egress ports 20 according to a selected service scheduling 
scheme such as weighted fair queuing (WFQ). When a queue/jmcket is serviced by one of the 
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arbiters 22, the corresponding memory block is freed, flie QMM 16 is notified as to which 
queue was serviced, and the packet is deUvered to the corresponding egress port 20 for 
transmission over an aggregate ou^t stream 21. 

The Venn and tree diagrams of Figs. 3 and 4 show how the physical 
memory 12 may be partiUoned in a hierarchical manner in accordance with the queuing 
scheme described with reference to Fig. 2. In this example, there are four levels in the 
hierarchical partitionment of memory 12. At a first or top level, the memory is logicaUy 
partitioned into a shared buffer space 14 which occupies a subset Oess dian or equal to) of the 
amount of fixed physical memory 12. The excess memory space above the shared buffer 
space represents free unallocated space. At a second level, the memory space allocated to the 
shared buffer 14 is partitioned amongst the various egress ports 20 of line card IIB. At a 
third level, the memory space aUocated to each egress port is further partitioned into service 
classes. At a fourth or bottom level, die memory space allocated to each service class is 
15 fiirther partitioned amongst individual traffic flows. In the case of ATM c<Hnmunications, 
these traffic flows may be individual VCs, as shown, such as virtual channel circuits (VCQ 
and virtual padi circuits (VPC), but in 25 otfier types of communication systems other types 
of logical packet streams wiU constitute traffic flows. For instance, in Internet Protocol (IP) 
networks label switched paths are a form of traffic flow. Similarly, the logical packet streams 
20 resulting from die forwarding rules of a packet ckissifier effectively constitute traffic flows. 

In general, at each level of die hierarchical partitionment of die memory 12 
there may exist one or more memory partitions, wherein each such partition is fiirther 
iaibdivided into one or more partitions, individually referred to herein as a "child" partition, 
25 located on an immediately lower level of the hierarchy. The exception to fliis, of course, 
occurs at die bottom-most fcsvel of the hierarchy wherein the memory partitions are not further 
subdivided. Similarly, a partition located at the top-most level of die hierarchy will not have 
a "parent" partition. 
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in the present application O c., communications systems), since each memory 
partition (e.g., shared buffer, ports, classes, and VCs) represents memory space notionally 
allocated to a group or set of one or more traffic flows at various levels of granularity, there 
also exists a corresponding traffic flow hierarchy. For instance, in the embodiment shown in 
5 Figs. 3 and 4, one fourth level traffic flow set consists of an individual VC 25a, and one 
second level traffic flow set consists of a group of VCs 2S^^\ including VC 25a, associated 
with egress port no. 1 (ref. no. 20a in Fig. 2). It will be understood from the present example 
that a given traffic flow set consists of one or more subset trafGc flows, individually referred 
to herein as a "child" set, located on an immediately lower level of the hierarchy. The 
10 exception to this occurs at the bottom-most level of the hierarchy wherein the traHic flow sets, 
consist of an individual (or aggregate) logical packet stream and will not have any defined 
subsets. Similarly, a traffic flow set located at the top-most level of the hierarchy will not 
have a "parent" set. 

15 The memory partitions are "soft" as opposed to "hard" partitions, meaning that 

if the memory space occupied by packets associated with a given partition exceeds the size of 
the partition then the QMM 16 does not automatically discard incoming packets associated 
with that partition. Rather, the size of a given partition can be viewed as a target memory 
occupancy size for the traffic flow set corresponding to that partition. At the lowest level of 

20 the hierarchy, however, the partition size functions as a threshold on the amount of memory 
that may be occupied by an individual traffic flow. When this threshold is exceeded, the 
QMM 16 enables packet discard. More specifically, the QMM may be configured to effect 
packet or cell discard, or to effect early frame or partial frame discard for frame based traffic. 

25 The size of each partition is generally variable and dynamically determined by 

the QMM 16 in order to control the aggregate congestion of the memory 12. More 
specifically, at each level of the hierarchy, the aggregate congestion within a given parent 
memory partition is controlled by computing a nominal partition size that can be applied to 
each of its child partitions (existing at the immediately next lower level). The value of the 
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nominal partition size for the child partitions of a common parent can be based on a number 
of factors such as the degree of congestion, its rate of change or even the mere existence or 
non-existence of congestion within die parent partition. Specific embodiments are given 
below. Regardless of the ftinction, tfus process is recursively carried out throughout the 
5 hierarchy in order to dynamically determine the size for each partition at each level of the 
hierarchy. In this manner, aggregate congestion at higher levels percolate down through the 
hierarchy to effect the memory occupancy thresholds for individual traffic flows. 

A second embodiment, implemented in software, is now described in order to 
illustrate the invention in operation. Referring additionally to Figs. S and 6, this more 
simplified embodiment is directed toward a single-port buffering subsystem 10 ' wherein the 
memory 12 is partitioned into a shared memory buffer 14 ' provisioned specifically for ATM 
ABR and UBR traffic. The remaining portion of the memory 12 may be aUocated to other 
ATM service categories, as described previously, or reserved for over-allocation putposcs. 
Fig. 6 shows the hierarchical partitionment of the memory using a tree structure. Since 
subsystem 10' features only one egress port, no provision has been made in this hierarchy for 
partitioning the memory amongst egress ports as in the previously discussed embodiment. 
Thus tiie hierarchically partitionment of the memory 12 and the corresponding traffic flow 
hierarchy features only tiuw levels, namely shared buffer 14% service categories 16, and 
VCs 25. 

The following pseudo-code demcmstrates the algorithm executed by QMM 16 in 
this embodiment. 



PSEUDO-CX)DE 

Variable DEnNmoN: 
Per Buffer 

• TBS - A constant which provides a target size for the buffer, in units of cells 
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• B_count - Counter for measuring the total number of cells stored in the buffer, 
thereby reflecting the amount of shared buffer currently being utilized. 

• Last_B_count - A variable for holding the measure of the total number of cells 
stored in the buffer during a previous iteration. 

• TSCS - A control variable which is used to set a target size (in terms of the 
number of cells) for a service class within the buffer. TSCS varies over time 
based on a disparity between TBS and B_count, as explained in greater detail 
below. 

• FBS - A constant used to provide a lower bound on TSCS. 

• Dl, D2 D3 and IM - Constants used to effect a geometric series or progression, 
as discussed in greater detail below. 

Per Service Class 

• SC_count [i] - Counter for measuring the number of ceUs in service class i, 
thereby reflecting the actual memory occupancy for the service class. 

• Last_SC_count [i] - A variable for holding the measure of the total number of 
cells in service class i during a previous iteration. 

• [i] - A constant used to specify a weight for service class i. 

• TVCSpl - A control variable which is used to set a target size for a connection 
within service class i. TVCSp] varies over time based on a disparity between 
TSCS*wsc[i] and SC_count[i], as explained in greater detail below. 

• TCSmin and TCSmax - Constants used to apply minimum and maximum 
constraints on the value of TVCS[i]. 

Per Connection 

• VC__count[i]|j] - Counter for measuring the number of cells stored for 
connection j of service class i. (Note that the number of connections associated 
with each service class may vary and hence j may correspondingly have a 
different range for each value of i.) 
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• MCRplDl - Constant indicative of the MCR or weight of VC j of service 
class i. 

• VCT[i][j] - Variable for the cell discard threshold for connection j of logical 
VC group i. The cell discard threshold is proportional to the corresponding 
TVCSp]; more specificaUy, VCT[i]D] = TVCS[i]*MCR[i]D]. 

Initialization: 

(100) TSCS := TBS*FBS 

(102) TVCS[i] : = 1 V i, i€ {1 . .N} where N is the number of service classes. 



Periodically Calculate TSCS: 

(104) if ( (B_count > TBS) & (B_coum > Last^B count) ) 

(106) TSCS:= TSCS*(1-D1) 

(108) else if (B_count < TBS) 

15 (110) TSCS:= TSCS/(1-D2) 

(112) end if 

(114) subject to constraint that TBS*FBS S TSCS ^ TBS 

(1 16) Last_B_Count : = B^count 

20 Periodically Calculate TVCSH (V 0: 

(118) if ((SC^countM > TSCS*Wsc[i])& (SC_count[i3 > Last_SC_count[i])) 

(120) TVCS[i]: = TVCS[i]*(l-D3) 

(122) elseif(SC_count[i] < TSCS*Wsc[i]) 

(124) TVCS[i]: = TVCS[i]/(l-D4) 

25 (126) end if 

(128) subject to constraint that TCSmin ^ TVCSp] ^ TCSmax 

(130) Last_SC_Count[i] := SC^countp] 
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Upon cell arrival for VC[i1[j]: 

(132) VCTHD] := TVCSP] * MCR[i][fl 

(134) if( VC_count[i][jl > VCT[i][j] ) 

(136) enable EPD 

(138) end if 



The algorithm involves dynamically computing a target memory occupancy size, i.e., memory 
10 partition size, for each traffic flow set. This is symbolized in Fig. 6 by the solid lines used to 
represent each entity. The actual amount of memory occupied by each traffic flow set is also 
measured by the algorithm and is symbolized in Fig. 6 by concentric stippled lines. Note that 
the actual size of memory occupied by any traffic flow set may be less than or greater than its 
target size. 

15 

The algorithm utilizes current and historical congestion information of a given 
memory partition/traffic flow set in order to determine tfie nominal target size for its child 
sets. Broadly speaking, the algorithm dynamically calculates for each traffic flow set 
(a) a target memory occupancy size, and 
20 (b) a control variable, which represents the nominal target memory occupancy size 

for the children sets of the present set. 
In the algorithm, which is recursive, die target memory occupancy size is calculated at 
step (a) for the present traffic flow set by multiplying the control variable computed by its 
parent by a predetermined weight or factor. These weights, provisioned per traffic flow set, 
25 enables each child set of a common parent to have a different target occupancy. 

The value of the control variable calculated at step (b) depends on the 
congestion of the present traffic flow set. In the algorithm, congestion is deemed to exist 
when the target memory occupancy size exceeds the actual memory occupancy size of a given 

20625744.1 



<CA__ 2273291A1 I > 



CA 02273291 1999-05-27 



-15- 

set. At each iteration of the algorithm, the value of the control variable is decxeased if 
congestion currently exists and if the traflGc flow set previously exhibited congestion. This 
historical congestion information is preferably based on the last iteration of tfie algorithm. 
Conversely, the value of the control variable increases if no congestion exists for die traffic 
5 flow set. Thus, in this embodiment, the target occupancy for the child sets of a common 
parent are based on a disparity between the target and actual memory occupancies of the 
parent. 

Steps (a) and (b) are performed for each traffic flow set at a particular level to 
10 calculate the respective target occupancies for the child sets diereof at the next lower level of 
the hierarchy. Anodier iteration of tiiese steps is performed at the next lower level, and so 
on, until target occupancies are calculated for the traffic flows at the bottom-most level of the 
hierarchy. 

15 For instance, the target occupancy for service classes 16A and 16B is based on 

a disparity 30 between the target and measured occupancy of shared buffer 14'. Similarly, 
the target occupancy for each VC 25^^'> to 25^^ is based on a disparity 34A between the 
target and measured occupancy of service class 16A. When an AAL frame or cell is 
received, the algorithm identifies the corresponding VC and determines whether its actual 

20 memory occupancy exceeds the target memory occupancy size thereof in which case the 
frame or cell is subject to discard. In diis manner congestion at higher levels of the traffic 
flow hierardiy percolates through the cascaded hierarchical structure to effect the tiiresholds 
of individual connections. 

25 Referring additionally to the pseudo-code, TBS represents the target memory 

occupancy size for buffer 14. TBS is a fixed value at the highest level. TSCS represents a 
nominal target size for aU service classes 16, and TSCS*Wsc[i] represents the target size for a 
particular service class. The factor Wscfi] is the weight applied to a particular service class in 
order to allow different classes to have various target occupancy sizes. Similarly, TVCSp] 
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represents a nominal target size for the VCs 25 within a iMirticular service class i, and 
TVCS[i]*MCR[i]D], which is equal to VCT[i]Ij], represents the target size, as wcU as the cell 
discard threshold, for a particular VC. The factor MCR[i]0] provides MCR proportional 
distribution of buflfer space within a service class. TSCS and the values for each TVCSp] and 
5 VCT[i][j]are periodically computed and thus will generally vary over time. 

A variety of counters (B Count, SC_Count [i], VC_Count [i][j]) are employed 
to measure the actual memory occupancy size of the various traffic flow sets. These are 
updated by the QMM 16 whenever a cell, i.e., a fixed size packet used in ATM systems, is 
10 stored or removed from buffer 14. (The updating of counters is not explicidy shown in the 
pseudo-code.) 

Lines 100-102 of the pseudo-code initialize TSCS and TVCS[i] V i. TSCS is 
initialized to a target size of TBS*FBS. FBS is preferably equal to 1/N, where N is the 
15 number of service classes 16 witfiin shared buffer 14'. This has the effect of initially 
apportioning the memory buffer equally amongst each service class. Other initialization 
values are also possible. TVCS[i] is initialized to 1 for each connection, as a matter of 
convenience. 

Lines 104 - 116 relate to the periodic calculation of TSCS. Line 104 tests 
whether the actual occupancy of shared buffer 14' is greater than its target occupancy and is 
increasing. If so then at line 106 TSCS is geometricaUy decreased by a fector of 1-Dl, where 
0< Dl < 1, e.g., 0.1. Line 108 tests whether the actual occupancy of shared buffer 14' is 
less than its target size. If so then at line 110 TSCS is geometricaUy increased by a factor of 
25 1/(1-D2), where 0< D2 <1 e.g., 0.05. The values of Dl and D2 are preferably selected 
such tfiat when the target occupancy decreases it does so at a fester rate than when it 
increases, as exemplified by the respective vaUies of 0.1 and 0.05. Those skilled in this art 
will appreciate that Dl and D2 control how fast the system responds to changes of state and 
that some degree of experimentation in the selection of suitable values for Dl and D2 may be 
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required for each particular application in order to find an optimal or critically damped 
response time therefor. 

Line 114 constrains TSCS to prescribed maximum and minimum limits of TBS 
5 and TBS*FB respectively. The maximum limit prevents service classes from attaining a 
target occupancy value beyond the availability of the shared buffer. The minimum limit 
bounds TSCS to ensure that it does not iterate to values that would cause convergence times to 
suffer. 

10 Lines 118 - 130 relate to the periodic calculation of TVCS[i] in relation to 

service class i. Line 118 tests whether the actual occupancy size of service class i is greater 
than its target size and is increasing. If so then at line 120 TVCSp] is geometrically 
decreased by a factor of 1-D3, uiiere 0 < D3 < 1, e.g., 0.1. Line 122 tests whether the 
actual size of service class i is less than its target size. If so dien at line 124 TVCSp] is 

15 geometricaUy increased by a factor of 1/(1-I>4), wiiere 0 < D4 < 1, e.g., 0.05. The values 
of D3 and D4 are preferably selected such that when TVCSp] decreases it does so at a fester 
rate dian when it increases, as exemplified by the respective values of 0.1 and 0.05. 

Line 128 constrains TVCSp] to prescribed maximum and minimum limits to 
20 ensure that convergence times are not excessive. TCSmax is preferably equal to TBS/LR, 
where LR is the line rate of the corresponding output port. This upper bound also ensures 
that a connection can never receive more than TBS buffer space. TCSmin is preferably equal 
to TBS/MCRmin, where MCRmin is the minimum MCR of all connections. This provides a 
conservative lower bound. 

25 

In this embodiment the QMM 16 effects early packet discard (EPD), and thus 
lines 132-138 are actuated when a start-of-packet (SOP) ceD is received by die QMM 16. (In 
the AAL5 ATM adaption layer protocol the end of packet (EOP) ceU signifies die start of the 
next packet) The target memory occupancy size or dueshold for VC j of service class i is 
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cvaluated at line 132 when a SOP cell is received. The threshold is equal to TVCSp] 
multiplied by the MCR of Ac coniiection. As mentioned earlier, this provides for MCR 
proportional distribution of the buffer space allotted to service class i. Line 134 tests whether 
the number of cells stored for VC j exceeds VCT[i]Ij], its target occupancy. If so, then EPD 
5 is enabled at line 136 and the QMM 16 subsequently discards all cells associated with the 
AAL5 frame. Lines 132 to 138 are re-executed upon the arrival of the next SOP cell. In the 
alternative, the system may effect a partial packet discard (PPD) policy. Alternatively stiU, 
line 136 may be modified to effect cell discard per se, with lines 132 - 138 being executed 
upon the arrival of each cell, 

10 

This embodiment is readily scalable to systems having a large number of 
service classes and connections since there arc relatively few state variables associated with 
the shared buffer and the service classes. In addition, most computations may be performed 
in the background, thereby minimizing processing requirements on time critical cell arrival. 

15 

This embodiment also allows fiiU buffer sharing. To see why this is so, 
consider an extreme case where all VCs associated with service class 16B cease transmitting 
cells. In this case, the shared buffer 14' begins to rapidly empty, causing the measured 
buffer size to be significandy smaller than the target buffer size. This causes the target sizes 

20 for service classes 16A and 16B to increase up to a level of TBS, the target size of the buffer. 
In turn, TVCSp] for all connections rises to an amount which enables the service category 
occupancy to reach TBS. Consequendy, die entire buffer becomes available to all of the 
transmitting connections of service class 16A and fuU buffer sharing is achieved. Moreover, 
it wiU be noted that each VC 25^'> to 25*^> of service class 16A receives a share of the 

25 buffer space allotted to that service class in proportion to the MCR of die connection. 
Consequendy, the instantaneously unused buffer space of service class 16A is distributed in 
proportion to the MCRs of the connections within the service class. 
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Thc method of allocating buffer space has been particularly described with 
reference to the three level traffic flow hierarchy as shown in Fig. 6. Those skiUcd in the art 
will understand that the method can be applied with respect to an n-level traffic flow 
hierarchy. 

5 

For example. Fig. 7 shows a four level hierarchy wherein the memory 12 of the 
second embodiment is partitioned amongst multiple egress ports 20. The level of the port 
partitions are disposed between tfie levels for the shared buffer 14 ' and service classes 16. In 
diis hierarchy, the target memory occupancy size for each port 20 is based on the disparity 30 

10 between the target and measured memory occupancy sizes of shared buffer 14 aiKJ the target 
sizes for the service classes 16 associated with a given port are based on a disparity 32A or 
32B between target and measured memory occupancy sizes of the given port. More 
specifically, let g{x,y} represent a discrete or iterative function wherein if x>y and x is 
increasing then g{x,y} geometricaUy decreases and if x<y then g{x,y} increases. The 

15 nominal target occupancy sizes for the various entities in the hierarchy shown in Fig. 3 can 
be: 



TBS = constant, 

TPS = g{B_^count, TBS}, 

TSCS[i] = g{P,count{i], Wp[il*TPS}, 

TVCS[iJ] = g{SC.count[iJ], Wsc[iJ]*TSCS[i]}, and 

VCT[ij,k] = TVCS[iJ]*MCR[ij.k]. 



In the foregoing, TPS represents a nominal memory occupancy for ports and 
Wp[i] is a weight associated with each port i. The product Wp[i]*TPS represents the target size 
for each particular port, which need not be equal. Similarly, Wsc P. j]* TSCS[i] represents 
the target size for a particular service class j associated with port i. 
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It should also be noted that g{x,y} may alternatively provide progressions other 
than geometric, including but not limited to linear, hyperbolic, logarithmic or decaying 
exponential progressions. Each type of progression wiU provide different convergence 
characteristics. Also, g{x,y} need not necessarily consider historical congestion information. 

5 

For example. Figs. 8-10 show anodier embodiment of the invention, 
implemented in hardware, which only considers current congestion. This embodiment is 
directed toward a buffering subsystem 10" wherein die physical memory 12 is partitioned into 
a shared memory buffer 14" provisioned for only one of ABR and UBR traffic, or 

10 alternatively for traffic from both classes. The remaining portion of the memory 12 may be 
allocated to other ATM service categories, as described previously, or reserved for over- 
allocation purposes. Fig. 10 is a tree diagram showing the hierarchical partitionment of the 
memory for diis queuing scheme. Since the subsystem 10" features only one egress port and 
no partitionment amongst service classes, the memory partitionment and corresponding traffic 

15 flow hierarchy only has two levels, namely shared buffer 14" and VCs 25. 

Fig. 9 shows hardware 40 incorporated within the QMM 16 of this embodiment 
for determining whether to enable or disable packet discard. The hardware 40 comprises 
three inputs, as follows: 

20 

Qs: A counter in respect of the total number of cells occupying the 

shared buffer 14", thereby reflecting the actual occupancy size of 
the shared buffer. This counter is incremented/decremented by 
the QMM 16 upon cell arrival/departure. 

VC-Count j: A counter in respect of die total number of ceUs occupied by VC 
j. This counter is incremented/decremented by the QMM 16 
upon the arrival/departure a cell belonging to VC j. 

MCR j: The MCR value of VC j. 
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The QMM 16 utilizes the hardware 40 whenever an end of packet cell (of an 
AAL frame) arrives, in which case congestion control is executed. The Qs counter or 
variable is fed to a quantizing function 42 which produces a quantized congestion variable CS- 
Qs, having a pre-spedfied range of values, e.g., 0 to 2047 (i.e., an 11 bit quantity). The 
5 quantization function maps Qs to CS_Qs based on tfie line rate of the egress port 20, For 
example, for a given value of Qs, an egress port having a line rate of 1 .6 Mb/s will map onto 
a lower quantized value CS_Qs than an egress jxjrt having a line rate of 800 kb/s. Table 1 
below shows an example of this mapping for some common standardized line rales where the 
pre-provisioned target size for the shared buffer 14" is 32k cells. 



Qs 


CS_Qs 


DS-3/B-3 


QC-3 


OC-12 


[12284, 32k] 


[28664, 32k] 


[24568, 32k] 


2047 


ooo 


ooo 


ooo 


o 
o 
o 


[4100,4103] 


[12296,12303] 


[8200.8207] 


1 


[0.4099] 


[0,12295] 


[0,8199] 


0 



Table 1 



It should be appreciated that CS_Qs thus corresponds to a disparity between the 
target and actual memory occupancy of the shared bu%r 14". It shouW also be noted that the 
15 function which measures congestion differs depending on the line rate of the egress port. 

The target memory occupancy or threshold, VCT, for a connection j featuring 
an EOP cell is computed by multiplying the MCR of the connection by a predetermined vahie 
selected from a lookup table 44 based on the quantized shared buffer congestion variable 
20 CS_Qs. The lookup table 44 provides in effect pre-computed values of a pre-determined 
function. Table 2 shows an example of such a pre-determined ftmction in respect of an OC- 
12 egress port. 
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Decimal Value of CS_Qs 
(Input) 


VCT 
(Output) 


[0, 488] 


MCR^l 20.96 


[489. 1697] 


MC1Vc120.96jiO.9926O94*°-*^> 


[1698. 2,047] 


0 



Table 2 



This table provides a decaying exponential function >*1ien CS-Qs is in the range 
of 489 - 1697; a maximum value of 120.96 when CS^Qs is m the range of 0-488, wherein the 
shared buffer is relatively uncongested; and a minimum value of 0 when CS_Qs is in the 
range of 1698 - 2047, wherein the shared buffer is deemed to be very congested. 

When the end of packet cell arrives, a comparator 46 compares the memory 
occupancy threshold of the VC, i.e., VCT, against VC_count[ j], and if the latter is greater 
than the fonncr an EPD signal 48 is enabled. Othenvise the EOP cell is stored and the EPD 
signal 48 is disabled. 

Those skilled in the art will understand that while the embodiments described 
herein have disclosed two, three and four level memory partition/traffic flow hierarchies, fer 
more elaborate hierarchies may be constructed. Other possible hierarchies include (from top 
level to bottom level): 

buffer, port, service category, groups of virtual circuits, individual 
virtual circuits; 

buffer, port, service category, queue, virtual circuit; 
buffer, port, service category, virtual path aggregation (VPA), and 
virtual circuit; 
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• buffer, port, service category, virtual private network (VPN), and virtual 
circuit; 

• buffer, port, service category, VPN, VPA. and virtual circuit. 

5 Similariy, those skilled in tfie art will appreciate that numercHis modifications and variations 
may be made to the preferred embodiment without departing from the spirit of the invention. 
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Claims 

1 . A method of partitioning a memory buffer, comprising: 

(a) defining a hierarchy of memory partitions, said hierarchy including at least a 
5 top level and a bottom level, wherein each non-bottom level memory partition 

consists of one or more child memory partitions; 

(b) provisioning the size of each top-level memory partition; 

(c) dynamicaUy computing a nominal partition size for the child partitions of a 
given non-bottom level memory partition based on the congestion of the given 

10 memory partition; 

(d) dynamicaUy computing the size of each cWld memory partition based on a pre- 
provisioned weight of the nominal partition size computed in step (c); and 

(e) iterating steps (c) and (d) in order to dynamicaUy determine the size of each 
memory partition at each level of the hierarchy; 

wherein the memory partitions at tfie bottom-most level of the hierarchy represent 
memory space aUocated to a traffic flow and the size of the bottom level partition 
represents a memory occupancy threshold for the corresponding traffic flow. 



15 



2. 

20 



3. 



25 



The method according to claim 1. wherein the nominal partition size for a given group 
of child memory partitions increases if the parent memory partition is not congested 
and decreases if the parent memory partition is congested. 

The method according to claim 1, wherein each memory partition is associated with a 
preKietermined coUection of packets and the congestion of a given memory partition is 
correlated to a disparity between the size of the given memory partition and the actual 
memory occupancy size of the corresponding collection of packets. 
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4. The method according to claim 1. wherein the congestion of a given memoty partition 
for the purpose of computing a nominal partition size in stq) (c) is based on current 
and historical congestion of die given memory partition. 

5 5. A metiiod of buffering packets in a communications device, the method comprising: 

(a) defining a hierarchy of traffic flow sets, wherein the hierarchy includes at least 
a top level and a bottom level, and each non-tottom level traffic flow set 
comprises one or more child traffic flow subsets; 

(b) provisioning a target memory occupancy size for each top-level traffic flow set; 

('^^ dynamically determining a target memory occupancy size for each traffic flow 

set having a parent traffic flow set based on a congestion of die parent traffic 
flow set; 

(d) measuring die actual amount of memory occupied by die packets associated 
widi each bottom level traffic flow; and 
15 (e) enabling die discard of packets associated widi a given bottom level traffic flow 

set in die event die actual memory occupancy size of die corresponding bottom 
level traffic flow exceeds die target memory occupancy size tiiereof. 



6. The metiiod according to claim 5, wherein each non-top level traffic flow set is a 
subset of a traffic flow set located on an immediately higher level of die hierarchy and 
each non-bottom level traffic flow set is a superset of at least one traffic flow set 
located on an immediately lower level of die hierarchy 

7. The mediod according to claim 6, including measuring die actual amouM of memory 
25 occupied by each traffic flow set of die hierarchy . 



The mediod according to claim 7. wherein step 5(c) inchides computing a nominal 
target occupancy size for all die child traffic flow sets of a common parent and 
provisioning each such chad traffic flow set widi a weight, wherein die target memory 
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10 



occupancy size of each such chUd traffic flow set is a weighted amount of the nominal 
target occupancy size. 

9. The method according to claim 8. wherein the nominal target occupancy size for a 
group of child traffic flow sets having a common parent changes in accordance with a 
prespecified function in response to the congestion of the common parent traffic flow 
set. 

10. The method according to claim 9, wherein congestion is correlated to a disparity 
between the target and measured memory occupancy sizes of a traffic flow set. 

11. The method according to claim 7, wherein the target memory occupancy size for each 
non-top level traffic flow set changes in accordance wddi a prespecified fimction in 
response to a disparity between the target and measured memory occupancy sizes of its 

IS parent traffic flow set. 

12. The method according to claim 11, wherein the target memory occupancy size for a 
non-top level traffic flow is limited to a prespecified range. 

20 13. The method according to claim 12, wherein the target memory occupancy size for a 
non-top level traffic flow set 

(a) decreases iq> to a minimum limit if the measured monoiy occupancy size of 
its parent traffic flow set is increasing and exceeds the target size thereof, 
and 

2^ (b) increases to a maximum limU if die actual memory occupancy size of its 

parent traffic flow set is less than the target size thereof. 

14. The method according to claim 5, wherein a bottom level traffic flow set comprises an 
individual traffic flow selected from die group consisting of: a virtual connection; a 
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label switched path; and a logical stream of packets resulting from the forwarding niles 
of a packet classifier. 



A method of processing packets at a queuing point in a communications device, tf»e 
method comprising: 

receiving and associating each packet with a traflfic flow, said traffic flow being 
associated with a bottom level in a multi-leveled hierarchy of traffic flow 
sets also having a top tevel. wherein each non-top level traffic flow set is 
a subset of a traffic flow set located on a higher level of the hierarchy 
and each non-bottom level traffic flow set is a superset of at least one 
traffic flow set located on a lower level of the hierarchy; 

dynamically determining a target memory occupancy size for each non-top level 
traffic flow based on congestion of the corresponding traffic flow 
superset, wherein the target memory occupancy size is pre-specified for 
each top-level traffic flow set; 

and subjecting a given packet to the possibiUty of discard in the event the 
measured memory occupancy size of die corresponding traffic flow 
exceeds the target memory occupancy size thereof and storing the given 
packet otherwise. 

The method according to claim 15, wherein the target memory occupancy size for a 
non-top level traffic flow changes in accordance with a prespecified function in 
response to a disparity between the target and measured memory occupancy sizes of 
the corresponding traffic flow superset 

The method according to claim 16. wherein the target memory occupancy size for each 
non-top level traffic flow is limited to a pixsspecified range. 
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The method according to claim 17, wherein the target memory occupancy size for a 
given non-top level traffic flow 

(a) decreases up to a minimum limit if the measured memory occupancy size 
of the corresponding traffic flow superset is increasing and exceeds its 
target size, and 

(b) increases to a maximum limit if the actual memory occupancy size of the 
corresponding traffic flow superset is less than its target size. 

The method according to claim 15, wherein a traffic flow consists of one of; a virtual 
connection; a label switched path; and a logical stream of packets resulting from 
forwarding rules in a packet classifier. 

A method of buffering packets in a connection-orientated communications device, the 
method comprising: 

deiining plural partitions in a memory buffer and associating each connection 

which uses the buffer with one of the memory partitions; 
providing the buffer with a target memory occupancy; 

dynamically determining a target memory occupancy for each memory partition 
based on a disparity between the actual memory occupancy and the 
target memory occupancy of the buffer; 

dynamically determining a target memory occupancy for each connection based 
on a disparity between the actual memory occupancy and the target 
memory occupancy of the corresponding memory partition; and 

enabling packet discard in the event the actual memory occupancy of a 
connection exceeds its target memory occupancy. 

The method according to claim 20, wherein the target memory occupancy size for a 
given memory partition decreases up to a minimum limit if the actual memory 
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occupancy of the buffer exceeds its target occupancy and increases to a maximum limit 
if the actual memory occupancy of the buffer is less than its target occupancy. 

22. The method according to claim 21, wherein the target memory occupancy for a given 
S connection decreases up to a minimum limit if the actual memory occupancy of the 

corresponding memory partition exceeds its target occupancy and increases to a 
maximum limit if the actual memory occupancy of the corresponding memory partition 
is less than its target occupancy. 

10 23. The method according to claim 22, wherein said device employs an as3mchronous 
transfer (ATM) mode communications protocol and said packets are ATM cells. 

24. The method according to claim 23, wherein the actual memory occupancy of a 
connection is compared against its target occupancy immediately prior to or upon 

15 reception of die first cell of an ATM adaption layer (AAL) frame and early frame 

discard is effected in accordance with die outcome of the comparison. 

25. The method according to claim 24, wherein the ATM cells are associated with ATM 
service categories which specify a minimum cell rate ^CR) traffic characteristic, and 

20 the target memory occupancy for a given connection is weighted by its MCR within 

the corresponding memory partition. 
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